Bayesian Networks in Educational Assessment by Russell G. Almond Robert J. Mislevy Linda S. Steinberg Duanli Yan & David M. Williamson

Author:Russell G. Almond, Robert J. Mislevy, Linda S. Steinberg, Duanli Yan & David M. Williamson
Language: eng
Format: epub
Publisher: Springer New York, New York, NY

(10.5)

Correspondingly, the posterior predictive distribution for a fit index, e.g., Weaver’s surprise index for Task j, W j , is obtained as

(10.6)

The idea is to repeatedly draw shadow data sets y rep from a predictive distribution for data, using the posterior distribution of ω given the observed data y. In each such replicate, calculate the value of some statistic or index of interest. The resulting distribution is used as a reference distribution to evaluate the value calculated with y itself. (Exercise 10.11 is a simple problem the reader can do by hand to get a feel for PPMC.)

Although can be difficult to derive analytically in more complex problems, it is actually straightforward to sample from, especially if an MCMC algorithm was produced to sample from . In each cycle (or in selected cycles) of the MCMC loop, after the values are drawn for the parameters ω, a shadow data set y rep is drawn.

For instance, in the running latent class example in Chap. 9, for the first two tasks the probability of a correct response to Task j by a learner in Class k is Bernoulli(π jk ).We can, thus, write the probability for Learner i as π j,class(i). When the model is described in the BUGS language, the corresponding line in the model code is

y[i,j] ∼ dbern(pi[class[i],j]).

In every MCMC cycle, the observed value of y ij contributes to the likelihood function for π j0 and π j1 and all of the other unobserved variables in the model; in turn, values for each of them are drawn from their full conditionals as described in Chap. 9. To obtain a shadow draw for y ij in each cycle, we merely need to add the line

yrep[i,j] ∼ dbern(pi[class[i],j]).

Now in every cycle, the variable is part of the model. No value is observed for it so the sampler draws a value from its full conditional—which is exactly the same in form as the distribution for y ij . The draw is carried out with class(i) and π j,class(i) fixed at the values drawn for them this cycle. These values will vary from one cycle to the next. The posterior distribution for , thus, properly takes into account uncertainty about these variables, and all the other unknown variables in the problem.

Any descriptive statistic or analysis that can be run on y can also be run on the shadow data set y rep(t) from MCMC cycle t, and the results compared. Are there far too many zeros for Task j? Does a factor analysis of y yield factors that y rep(t) does not? In diagnostic testing, an interesting statistic is the number right on a subscale. Multiply the outcome vector by one column of a Q-Matrix to get a number right score focused on one particular skill. This will provide a measure of how well the model predicts performance on tasks requiring that skill. The range of features to compare which might shed light on model fit and model improvement is limited only by the analyst’s ingenuity.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.